add benchmark configuration and case for codeflash #1254
Conversation
|
Hi @aseembits93, is this ready for review? |
|
Hi @grzegorz-roboflow ! it's ready for review, please let me know any feedback, the feedback is invaluable to us. |
grzegorz-roboflow
requested review from
PawelPeczek-Roboflow,
hansent,
probicheaux and
yeldarby
as code owners
|
we are looking at resolving the failing tests |
|
|
||
| model = get_model(model_id="yolov8n-640", api_key=None) | ||
|
|
||
| benchmark(model.infer, images) No newline at end of file |
There was a problem hiding this comment.
is this symbol imported?
There was a problem hiding this comment.
Hi @PawelPeczek-Roboflow , We follow the style of pytest-benchmark in writing tests (https://pytest-benchmark.readthedocs.io/en/latest/) but override the benchmark keyword to suit our style of benchmarking. The 'benchmark' keyword is injected by codeflash during execution. This file is only executed by codeflash during codeflash --benchmark, it is outside any test subdirectory and won't be encountered by any of the CI workflows except the codeflash workflow.
5 participants
Hi @grzegorz-roboflow,
This Pull Request sets up a Codeflash benchmark optimization workflow that automatically optimizes code in future pull requests that modify benchmark code.
When Codeflash discovers optimizations, it guarantees their correctness and reports the expected performance improvements for each benchmark.
We've set up the following inference benchmark to be optimized in CI:
inference benchmark python-package-speed -m yolov8n-seg-640 -bi 10000We believe Roboflow would benefit from a broader set of end-to-end benchmarks that reflect real workflows.
To support this initiative, we'd like Roboflow to define benchmarks for workflows you consider important to optimize.
These benchmarks should be lightweight enough to run quickly and follow the pytest-benchmark format. (Note: Codeflash uses its own pytest plugin rather than the actual pytest-benchmark plugin.)
Let us know if you have any questions or need help setting this up!